Hi Cardiff!

• I’m Jasmine Dumas
  • Insurance Data Scientist and useR living in Connecticut, USA
• Slides and Materials available at: jasdumas/talks/caRdiff-uk-open-gov-beer

The search for datasets

• Finding interesting and available datasets is a concern for all those involved in data science.
• Where to look?
  • google
  • github
  • kaggle
  • open government data portals
  • machine learning repositories at Universities

The search continues for analysis-ready datasets

• If cleaning data is 70% of the data science process than the desire for analysis-ready data (even at the basic csv format) is priority #1 after addressing the business questions.
• analysis-ready data also needs to include some type of documentation

The U.S. Government open data portal

The “clearinghouse” for open U.S. government data is located at data.gov. It also contains tools, and resources to conduct research, develop web and mobile applications, design data visualizations.

Good examples of datasets

• “Good” datasets are “Good” if they can be read in to R with functions like: read.csv, read.table or even webscraped efficiently
• “Great” datasets take minimal cleaning aside from manipulation

##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 1          5.1         3.5          1.4         0.2  setosa
## 2          4.9         3.0          1.4         0.2  setosa
## 3          4.7         3.2          1.3         0.2  setosa
## 4          4.6         3.1          1.5         0.2  setosa
## 5          5.0         3.6          1.4         0.2  setosa
## 6          5.4         3.9          1.7         0.4  setosa

Bad examples of datasts

• “Bad” datasets are “bad” if they are in inconsistent formats, have mixed columns
• “Bad” difficult for analysis

The landscape of beer datasets

• There are limited amounts of datasets for beer analytics and the API’s that are availabe have restrictions on usage or require a user account
  • datasets: http://www.craftbeeranalytics.com/beer-data.html, SNAP-RateBeer* - which is provided to academic researchers only
  • api’s: RateBeer, Untapped, The Beer Mapping Project, BreweryDB
  • data analysis projects of beer: yhat BeerAdvocate Recommender

The reality and struggle of limited beer datasets

• … are real and lead to:
  • frustration
  • losing interest and passion for the domain
  • choosing another dataset
  • abandoning the data analysis project
  • breaking Terms of Service (ToS) with web-scraping

The sparcity of beer data actaully inspired me become better at finding datasets and webscraping!

• Here is a tutorial I wrote to get meta data from Beer Advocate using Import.io

ttbbeer

• An R data package of beer statistics from U.S. Department of the Treasury, Alcohol and Tobacco Tax and Trade Bureau (TTB)
  • The plan: liberate more beer statistics datasets from open U.S. government data portals to analysis-ready data frames for R and beyond
  • The method: web-scraping with rvest (the first dataset was copy/pasted to excel due to the limited R ecosystem of PDF parsing pacakges)
  • The dream: increase awareness of beer analytics and promote analysis-ready datasets from data.gov
• available on CRAN and Github

library(ttbbeer)
data("beermaterials")
print(head(beermaterials))

##      Month Year Malt_and_malt_products Corn_and _corn_products
## 1  January 2015              316756669                60186876
## 2 February 2015              294630835                49718568
## 3    March 2015              320802302                53925263
## 4    April 2015              312758553                51063078
## 5      May 2015              333902533                59741914
## 6     June 2015              385603072                59298220
##   Rice_and_rice_products Barley_and_barley_products
## 1               52438465                    8492680
## 2               48975137                    9355462
## 3               48710754                   22213428
## 4               55895095                   10015583
## 5               59210512                   11478057
## 6               47366385                   25372526
##   Wheat_and_wheat_products Sugar_and_syrups Hops_dry Hops_extracts
## 1                  1392183         72069615  1916516        298514
## 2                  2145359         74431855  1775144        298894
## 3                  3761816         82679762  3750830        321462
## 4                  2575534         83137106  2196140        321444
## 5                  3292994         81360042  2200520        345805
## 6                  4491138         78767085  3011107        403948
##      Other
## 1 11163548
## 2 12017223
## 3 13892190
## 4 13491920
## 5 15579095
## 6 33045174

Questions & Cheers!

+